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ABSTRACT 

This report synthesizes findings of five conferences 
funded by the National Institute of Education (NIE) to plan new 
Follow Through research. In particular, conference discussions 
focused on the notion of systematic 'chanqe; time-on-task as the most 
promising strategy for success; and encouragement of principal and 
teacher support for implementing program models. It was suggested 
that committed teachers are needed to make a program successful, and 
ways were suggested to "court" them. In addition, it was felt that 
efforts to strengthen the instructional process must address, among 
other subjects: (1) staff attitudes and expectations; (2) 
organizational structure and procedures, including student/staff 
assignments; (3) systems of incentives and rewards; (4) commitment of 
local boards; (5) responsiveness to locally demonstrated need; (6) 
the delivery system; and (7) the management syt^tem. in response to a 
discussion of problems in program implementation and replication, an 
entirely new format for subsequent Follow Through projects/research 
was suggested. This format would involve study of exemplary programs 
to lead to a verifiable theory of effective early primary education 
and would provide for the necessary adaptations to student 
differences and other system variables. A discussion of what 
administrators need to know about a program before it is adopted or 
adapted is followed by guidelines for new Fellow Through pilot 
projects. Also included are brief descriptions of the Detroit 
(Michigan), Napa (California), Oakland (California), and Cotopaxi 
(Colorado) pilot projects. Following a summary of what is known to 
date about Follow Through programs is a list of 40 background papers 
prepared for NIE. (AS) 
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V9hatever success this progran may achieve owes much to 
the field of coa^yensatory education out of which it grew. We 
at the Institute acknowledge the crucial input of literally 
scores of primary school educators, theorists, researchers, 
and evaluators throughout the country to its development. 
(The calibre of the professionals who participated in this 
process is indicated by the list of conference .participants 
on pages .) They talked, wrote, discussed, advised, and 

occasionally harangued (with good reason) . We listened. 

A "f ield-responsive* approach to desi< ^ ag a new pro~ 
graa is portrayed in this report. The sma^^ staff at the 
Institute who bore the ultimate responsibility for design- 
ing this progran Jeffiry Schiller, Thel Kocher, Xfarry 
Rudner, Hancy Borkow and myself — feel that, to paraphrase 
Churchill, "Never have so few owed so much to so many." 

Charles Stalford, 1982 
Project Coordinator ,NIE 



BACKGROUND: BUILDING ON PAST EFFORTS 



Over the past twenty years public and private fiinds have 
supported extensive research in primary education. Out of 
these studies have cone promising new directions £or improv- 
ing compensatory schooling. By applying what has been 
learned in these years from studying successful schools, new 
demonstration projects could point the way .towards more ef- 
fective education of disadvantaged children. However, this 
urgent task must now be accomplished under, new budget con- 
straints. 

. During the late 1960s, the Follow Through Program was 
initiated to make schooling more effective for low-income 
children by building, on the gains made in Head start. The 
methodology adopted, called Planned Variation, enabled a 
variety of educational alternatives to be demonstrated by 
model-sponsors with different philosophical and practical 
orientations. The evaluation of the results was conducted 
by the US Office of Education and its independent contractors. 

Although the research/evaluation aspects of Follow 
Tu.ough were criticized, the program gained respect for having 
provided over twenty alternative instructional models for the 
primary grades. Follow Thrc^gh provided a wide rai^e of health 
social and other support services to disadvantaged pupils, in- 
volving their i»rents in the schools and sometimes even en- 
couraging the parents themselv€^ to seek productive careers. 
(An extensive discussion of how the Follow Through programs 
were developed, and how they fared, is to be found in Follw 
Throaqhi Forces for Change in the Primary Schools , The High 
Scope Press, 1980) 



In the same years Follow Through (FT) was providing 
high qualitry, if expensive, programs, NIE, as the research arm 
of the federal Department of Education, was investigating the 
practice of compensatory education. This research spanned 
the elementary and secondary grades, but was predominantly 
focused on the former. 

Nimierous studies sponsored by NIE, FT and others over a 
decade indicate that active learning time (ALT) is the most 
powerful and direct determinant of achievement . 1 (In a com- 
parison of FT programs, the achievement gains produced by 
the Direct Instruction Model and the Behavior Analysis Model 
v^re consistently greater than those of other models.) 

The theoretical basis of ALT has been traced back to the 
studies of John Carroll in the early 1960s and Benjamin 
Bloom in the mid 1970s . Carroll formalized the notion that 
active learning time is solely responsible for learning and 
that the amount of it needed to accomplish learning is de- 
pendent on an individual's capabilities. 2 Bloom built on 
Carroll's model. In his discussion of Mastery Learning he 
confirmed that simple allocations of the same amount of time 
to each pupil will not bring about mastery — slower stu- 
dents need a good deal more time than quicker students. And, 
according to Bloom, both slow and quick students need Cues, 
Reinforcement (praise). Active Participation and Feedback.-^ 



An excellent review of the literature can be found in Harnisch- 
feger, Annegret, "Active Learning Time: Its Determinants and Its Role in 
School Learning," November 198I, The Beacon Institute, Inc., Kenilvforth, 
Illinois 600^3- The paper was cofreniss ioned by NIE. 

Carroll, J.B., "A Model for School Learning." Teachers Col lege 
Record , 6^, I963. 

"The Prediction of Success on Intensive Foreign 
Language Training." In: Glaser, R. (Ed.) Training and Education , 
Pittsburgh, PA; University of Pittsburgh Press, 1962. 

^Bloom, B.S., Human Characteristics and School Learning , N.Y., 
McGraw-Hill, 1976. 

"Time and Learning," American Psychologist , 29 , 197^- 

"Thought Processes i' Lectures and Discussions," 
Journal of General Education, 7, 1953.' 
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These studies themselves can be seen as part of a longer 
tradition. Folk wisdom reminds us that practice makes per- 
fect, and educational theorists, including Dewey and Montes- 
sori, recognized the need to engage children actively in 
their learning. A curriculum tailored to engage the active 
interest and meet the skill level of each child also lay be- 
hind such diverse styles of instruction as Individualized 
Education, Open Education, Mastery Learning, and Direct 
« Instruction. 

Buc while the individualized programs recognized the 
importance of student engagement and accommodated by en~ 
♦ couraging children to select the task and determine how long 

they would spend at it — the programs did not deal with two 
significant findings of the 6-year Beginning Teachers Evalu- 
ation Study (BTES) . In tMs NIE-funded study of over 300 
experienced teachers, it was found that children stayed more 
on task when working directly with a teacher and made greater 
gains when working at a high success level. 

Exceptionally skilled teachers were able to allot a 
substantial amount of time for academics, present them in 
a lively manner so as to engage a diverse group, and simul- 
taneously meet each individual's need for success and re- 
inforcement. And the achievement of these children reflect- 
ed it. 

Naturally, most teachers are not that skilled. Many 
waste valuable teaching time on unproductive activity. Scxne 
can't maintain sufficient order, others wander off the sub- 
ject matter or spend r.oo much time in transitions or admin- 
istrative paperwork. Many aim their lessons too high or too 
low for the children. 

However, research sponsored by the NIE showed that with 
proper support, most teachers could be trained to be more 
effective. Improved classroom management procedures could 
help them carve out more quality academic learning time — 
4 which would raise the achievement scores. 



New Programs 

In June, 1980, NIE was invited by the Office of Elemen- 
tary and Secondary EducatJbn in the Department of Education 
to mount research which would utilize a portion of the Fol- 
low Through (FT) funds specifically set aside for research 
and development. Of the full FT budget for 1981, NIE was 
allotted $2.5 million (approximately 5,6% of th«2 total 
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FT budget for the year and 30% of the FT research budget) to 
fund pilot projects which would help improve P^i^^^Y school 
compensatory education programs. Due to reductions the FT 
pro-am suffered subsequently in budget, NIE was finally allot- 
ted $700,000 in 1981, with $1.2 million additional to be allot- 
ted ii« 1982 ai^ 1983. 

This money gave the NIE the opportunity to build new 
programs on successful research in the past decade. In 
addition to the BTES and the Direct Instructional Model 
developed by the University of Illinois for Follow Through, 
NIE wantea to follow up the More Effective Schools studies 
which connected better classroom, work to more supportive 
school management and leadership. 

The money also gave the Institute a chance to listen to 
the field. To gat a synthesis of the field's thinking, NIE 
commissioned 44 papers and invited over 180 educational 
theorists, researchers. Follow Through sponsors and national 
and local educators to five conferences, at which experts 
met in groups to discuss desirable directions for the new 
projects- While the Institute was determined to consider 
the recoranendations made by the conference participants, it 
did not intend to pursue research that was either of little 
current value, or that was repetitive. And, aware that 
schools* funds would be considerably reduced in the coining 
years, the Institute was interested in testing programs which 
would be not only effective but econcsnical and replicable. 
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The syntheses of these meetings are available. Contact 
Charles Stafford at the WfE and request "Planning for 
Iwproved Prtmarv School Education," 1982 

Portlmid, Oregon (February li-5, I!J8|) 

Thirty-eight laostlr *<est-coast educators, manv of whan were associated 
with Follow Through programs, attended. Among the participants who 
discussed deslreaill^ characteristics for the new projects vfere: 

Douglas Carnlne, University of Oregon (keynote speaker) 
JdM Evans, Educational Testing Service, Berkeley, CA 
Walter Hathaway, Portland Public Schools (who co-hosted the conference) 
Asa Hilllard, Georgia State University 

Preston Krcmkosky, Southwest Educational Oevelopreent Laboratory 
Will lam Spady, Anerlcan Association of School Administrators, Washington DC 
Robert Stahl, California Teachers Association 
Raywond Garza, University of California, Riverside 
Alice Paul, University of Arizona 
Robert Stahl, California Teachers Association 
Leslie R. Williams, Teachers College, 

Philadelphia. Pennsylvania (February 10-11, I98I) 

Fifty mostly east-coast educators, many of whom %^re associated with FT 
programs, attended. Among the participants who discussed deslreable 
characteristics for the new projects were: 

Robert Egbert, former Director of Follow Through 
Hortense Jones, New York City Board of Education 
Dick Jung, National Advisory Council 00 the Education of 

Dlsadvanti^ed Childr^ 
Pat Oifiistead, University of North Carolina (keynote speaker) 
Thomas HcNanara, Philadelphia Public Schools 
Lorraine Smlthberg, Bank Street FT Progran Director 
Barak Rosenshine, University of Illinois 
Linda Stebbins, Abt Associates, Cambridge, HA 
John Porter, Chief State School Officer , Michigan 
Hich;sel Keane, ETS , Evanston, IL (who returned to Phlla. to co-host 

« , , the conference) 

Gordon Kiopf, Bank Street College of Education, NY 
John Ogbu, University of Delaware 
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Austin. Texas (Febroary 19-20, 1981) 

Thirty-four evaluation specialists discussed field evaluation of the 
new projects. Attendees liKluded: 

Eva Baker p University of California at LA ^ * 

Jane David, Palo Alto t CA ' * 

HIchael Fullan» Ontario Instltlte 

Walter Hodgesr, Seor^la State University 

Richard Jaeger. University of North Carojfna 

Mary Kennedy, Huron instltlte, HA 

Stuart Rankin, Detroit Public Schools 

Robert St. Pierre* Abt Associates, Inc. 

isuara Santlago*Santlago, TC, Columbia University, HY 

Freda Hoi ley, Austin School District (who co-»K>sted the conference) 

Pittsburgh. Pennsylvania (March 12-13, 1^1) 

Thirty educational researchers and evaluation specialists discussed basic 
research needed to support the new projects. Attendees included: 

Ernest House, University of Illinois 

Walter Haney, Huron instltlte, MA 

E<ftmind Gordon, Yale Unvlverslty 

Thonaa HcNanara, Philadelphia Public Schools 

Margaret Wang, Unvlerslty of Pittsburgh (who co-hosted the conference) 

Gene Glass, University of Colorado 

David Welkart, High/Scope, Ypsllantl. Ml 

Edward ZIgler, Yale University 

Chad Eirett. University of Georgia 

Dalton Miller-Jones, University «^f Mass. Amherst 

Susan Loucks, The Netitfork, Inc, Andover, HA 



Washington, D.C. (November 16-17, 1982) 

To help the districts refine their winning proposals, representatives from 
the four pilot projects awarded NIE FT research contracts to Increase 
efficient use of learnlnq time, wet with technical advlsotrs (TAs) in eval- 
uatlcm and staff relatiws. Among those at the meeting were: 

9avld Derllner, NIE /University of Arizona (TA to projects) 
Gary Borlch, University of Texas (TA to projects) 
Sheldon Sofer. Detroit Public Schools (Project Director) 
/ Marilyn Jones .Oakland Public Schools (Project Director) 
Dennis Sparks, Consultant, Ml (TA to projects) 
Judith W. Little, Consultant (1^ to projects) 
Yvonna S. Lincoln, University of Kansas (TA to projects) 
Pan Robbins, Napa County, CA (Project Director) 
David M. Trujillo, Superintendent of Schools, Cotopaxl. CO 
Robert Floden, Michigan State Unvlerslty (TA to projects) 
Virginia Ko^ler, NIE 
Ursula Pllfero, NIE 

Larry VI Hers, Custer County School, WestcUffe, CO 
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THE CC»IFERENCES: TOWARD IMPROVED STRATEGIES 



Before the conferences, the research NIE foand most 
Dromlsinq («&ilc^ focused on noare effective manageaent c£ iitstructicn) 
had not explicitly addressed how the gnew practices could be 
implemented in schools on a systolic basis. For this rea- 
son, the Institute thought it best to focus the new programs 
on instructional management and iarolementation on a svstem- 
wide basis. ^ 



A. Initial Planniiig Ccoceiat 

The initial issue statopent participants at Portlanl azxs Pnila- 
delphia were asked by HIE representatives to consider and 
respond to read as follows: 

^ *Thc past ten years ">ave been enormous attention focused on 

the educational needs of low-Income children, and on how schools 
could better ^eet them. Hundreds of programs, many sponsored by 
the Federal governifient through Title I of the Elementary and Sec- 
ondary Education Act, have been developed at the local level. 
Many others have been tested systematically through the Follow 
Through Program. 

ffeich effort has been devoted to Identifying transferable 
► -r dge from these programs. The Department of Education's Joint 
nation Review Panel has validated over 300 exen^lary programs 
have emerged from Title I, Title IV-C, Follow Through and 
programs. Through the National Diffusion Network, over »20 of 
these are being disseminated to. schools In every state and territory. 



8 

Th« (^search and DevetofHi^t Exchange funded byN?E brings together 
craft and research knowledge on Improved techniques of classroom 
management 9 staff develofment, and teaching methcKis. Hany ^tate 
education agencies have ^opted IVD processes which are us^ to 
Idei^tlfy^ validate^ and disseminate exa^lary education practices 
within their state. The list of dtssemlnatlcMi efforts easily 
could be expanded. 

Mille no sifii|)1e recipes for success have emerged fr«n these 
efforts, they provide a rich body of suggestions abcnit N^w to make 
Follcw Through better. Rather than develop rww currlcular or 
learning theories armt In the first new Follow Through approaches 
NtE wants to exploit available Information. For exan^^l^^the 
demcmstrable successes of the Direct instructional ftaxJel and Be- 
havior Analysis Model In the Follow Through Natlmal Longitudinal 
Study lead many to concliKle that such approaches $>H>u1d be built 
upon. The success of former Is entirely congruent with a much 
1 . 7, r body of research on similar instructional strategies that 

to ^ch the same cwicluslcm. Hore recently the Beginning 
Teacher Evaluatlcm Study (6TES) has Identified the lQ4>ortance of 
"engaged academic learning time/* or tl^ on task, as an effective 
determinant of .earning. 

Somerset hidden In the overall conclusions of the variability 
within Individual FoIIqm Through models but still important are 
successes of numerous specific sites in individual iradels, analysis 
of wlilch can suggest* the viay to further In^r^vcmient. Included in 
these are models of "hi»ianl!5tic'' and cognltlvely oriented programs, 
those serving blllngtml chl Idren, and those systematically involving 
para:tSy fmillesp or c<»amjnlty In the child's education. 

With this rich body of information In hand, as weP as with 
respect for earlier diff Icul ties, NiE believes that the first wave 
of new pilot projects should not focus upon new curricula or 
instructional practices i^r se. Basic research and earlier exper** 
lences In Follow Through progr^ns Indicate that there are many 
instructional practices that can be effective^ if managed or imple- 
mented properly. Tim first new Follow Thrcxjgh approaches will 
therefore fc^s on demonstrating new ways in which LEA*s can over*- 
ccme barriers to effective instructional managenmnt and implcmienta- 
ti(xi. 

Illustrative thcmss ar<^nd v^ich pilot projects for the New 
Strand i approaches might be organized to use such knowledge include 

• Means to incree^^e instructional time In Follow Through 
Classrooms through In^roved managenmnt of services; 

m Itew patterns of staff develofment and selection of staff 

to gain better instructional management, including coopera- 

tlve>dgreements between schc^ls, teacher education Institu* 
tlons and teacher associations or unions; 
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-I • NeM way«> to systematically involve parent and cmcnunity 

groups in planning and conduc*- of FoUc^tf Through programs, 
incHuding the use of parents and famlHes to provide 
instruction in the home or conmunity; 

• New uses of information systems, including assessment and 
evaluation results, to bring better diagnostic and pre- 
script ive information to bear on Folloi^ Through student 

. learning needs; 

• New ways to facilitate support of school building and 
district administrators for substantial changes typically 

• required by innovative Follow Through procedures. 

Hit believes the new pilot projects should focus on systemic 
; change. That Is, they should not deal simply with changes In the 

classrocmi or other single coi^cHient. Rather, they should be 

- designed to bring about changes In the whole system required to 

- deliver instruction and should result in Increased coordination 

and support within that syst^. 

] Documentation and evaluation of the pilot projects will be of 

major importance. NIE expects to provide support to each pilot 
project to assist it In the design and execution of a documenta- 
tion and evaluation system. The system should help the project 
identify weaknesses so that these may be corrected and provide 

• information that external audiences Interested In using all or part 

of the project can use to determine its effectiveness. Of par- 

■t ticular interest in the documentation and evaluation system will be 

methods of determining hoM well the project is implenented. " 



Source : Plans for Follow Through Research and Development 

Jeffry Schiller, Charles Stalford, Lavnrence Rudner, 
Thel Kocher, Howard Lesnick. National Institute of 
Education Oct 1, 1980. 



B. " Systemic Change"; The Field Tries -t On for Size 

It v;asn't easy for most educators at the two meetings 
to get behind the notion of systolic change. Some worried 
that it was nebulous and needed more definition, others that it 
needed constraints, and still others asserted it was misguided 
because "effective change can only occur in small units. 

Their initial critical reactions may have been inf luencea 
by what Doug Cair.ine from the University of Oregon and the keynote 
speaker at the Portland conference called "a history of conflict in Follow 
Through." In bringing together old rivals, the Institute knew old ocfif licts were 
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bound to be aired? it also realized that the forthright ex- 
pression of concerns would presage prv/olems that could arise 
in the implementation of a new program. 

Since each pes: son had only to look across the table to 
find someone who held an opposite educational philosophy, it 
was easy for those affronted by what they saw to imagine a 
system-wide mandate that would outlaw or impede their own pre- 
ferred teaching style. For a while it seemed that the only 
agreement that would be reached in this group of truth-finders 
was that the person sitting acroiss from tliem at the conference 
table was misguided. Further, some suspected that this proj- 
ect was threatening the breath and life of what they held dear 
in Follow Through. They interpreted the NIE's focus on in- 
structional management to be a sign that the government was 
turning its back on parents, threatening to do away with aux- 
iliary services, giving priority to research over the needs of 
community and children, and putting an undue emphasis on 
achievement scores as the determining factor in program adop- 
tion. 

The conferees worried that system-wide change would ig- 
nore the needs and opinions of the teachers and. parents. They 
felt that better coordinaUon of federal and stat«k^rograms , 
research on maintaining and replicating a program iX.a state 
of high excitement and productivity were at least as important, 
if not more important, than systemic change. After hearing 
from the field , the Institute rethought the notion of 
systemic change. 

Despite NIE's decision, discussion on the ideal-si2e 
unit for change continued at subsequent conferences. Stuart 
Rankin, the assistant superintendent of Detroit's public 
schools, told the Austin evaluation conference that "the 
smaller the unit, the more amenable it is to change, and 
therefore,** according to Rankin, "system-wide change xs diffi- 
cult to effect.** 

Generally, school administrators were more aware and 
more sympathetic to the teachers' resistance to change than 
were academics. Dennis Sparks, an educational consultant from 
Dearborn, Michigan, speaking at the November Washington meet- 
ing, recalled how uncomfortable, angry and defensive he and 
other teachers had been when an administrator demanded that 
they adopt a new instructional system. 

"Rarely is the point made strongly enough, but change is 
always painful," agreed Sheldon Sofer, project director from 
the Detroit public schools. 
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Just as systemic change worried many of the Follow 
Through people at the first two conferences, classroc^ by 
classroom change was anathema to the evaluators. A classroom 
might be easier to change than a school, as Stuart Rankin 
claimed, but there were research problems connected with a 
one-class isla^id of change. 

The Austin conferees recommended that for research pur- 
poses all the K-3 classrocmis in selected schools should use 
the same method, and ideally two or three schools that dif- 
fered one from another in ethnic mix, size and location (rural 
or urban) should be involved. They hoped to sea material collected 
for at least five years, and with a caieful documentation of 
the schools* ecology (attitudes and school climate) have just 
the kind of data that would answer transferability questions. 
(Their suggestions were ultimately reflected" in tne desxgn 
of the RFP and helped guide the choice of districts selected.) 



C. Time-On-Task: The Most Promising Strategy 

"We have proven that dramatically higher basic skills' 
competency is produced with the direct instructional model," 
said John Evans of ETS, a le:»ding behaviorist. "If we set up 
a coordinated model, including the organization of instruction 
to increase time-on-task, and if we select and train teachers 
who use our tested and proven teaching methods faithfully, we 
can guarantee success. We have found the way to eliminate 
dysfunctional teaching styles. The method is teacher-proof. 
The kids all learn to read and compute." 

Not all Portland and Philadelphia participants were as 
enthusiastic as John Evans. Some vjere concerned that such an 
approach would dehximanize education. 

But ever, participants v:ho were not familiar vrlth the "Hodel 
of Classivom Instruction" (pictured on page 12) -om Carolyn 
Denharo's anthology Time to Learn ^ were aware that the bottom 
line was the achievement test scores . 



Fishery Charles W., Berliner, David C, et al. "Teaching Behaviors, 
Acadci-nic Learning Time and Student Achievement: An Overview," from Time To 
Learn edited by Denham and Lieberman. NIE 1980. 
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Flpne 1.1. A Modd of Cl^sioom instiucciafi 



Siodcsc 



envirofirncnT 



Icarnnif iu A««wa 
by lOMtctit tKha%1«) 



Achievemenf 



This enphasis on achiev^ent scores did not please 
those FT progrmn developers at the Philadelphia and Portland 
meetings who had rejected achievement scores as the propt^ 
gauge of their FT models in the past. 

But at the ffashisgton meeting, where David Berliner, 
the former director of the Far West Labs BTES study present- 
ed a comprehensive definition of engaged learning time, the 
enthusiasm was evident. The group meeting to refine their 
proposals for local implementation listened attentively as 
Berliner clarified the benefits and limitations of engaged 
learning time: 

increasing engaged tlm, which se^RS single, requires that 
Me isolvc rather difficult problans, like aastcrlng the delicate 
art of pacing, getting teachers to change their attitudes and 
their behaviors, and getting children to accept more responsibil- 
ity for their learning. 

Allocating more time for basic skills Is not the same as 
increasing engaged time. Even If a teacher set aside a whole 
day for basic skills work, the full time would not be utilized 

concentrated work. It couldn't be. So allocating time for 
^ work, while necessary. Is not sufficient. Even when one can 
make a nodest guess as to how much of the allocated time will be 
engaged time, we are not home free, increasing engaged time Is 
not enough. There are various levels of engagement. Children 
have to be challenged and, at the same tlii», be successful. 

According to Berliner, the six experts who wrXed on the 
BTES pooled their expertise and, after mulling, discussing and 
analyzing for six months, came up with the following formula- 
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tion: "The Engaged Learning Time which constitutes a part of 
the Allocated Learning '. Ime, should be divided again into 
three XK>re categories called haard (low success) , ipedium 
Cmedium success) , easy success. ** And, the BTES results indi- 
cate that students should be working on tasks that provide 
high success most of the time. 




But even that picture (as slicwn sSxr/e) isn't complete, for 
the student must be successfully engaged in learning which 
will be measured by an achievement test. 



As the diagram below shows, only where the outcomes that 
are tested overlap with the engaged learning time at a high 
success level do we have the most productive learning situa- 



HCMl/M SUCCESS 




Even those who feared teaching to the tests would 
cake time away from activities that prcanoted creative 
thinking, problem solving, enthusia^ for work and pro-active, 
cooperative student behavior were willing to support an effort 
to increase engaged time on meaningful tasks. While they 
still felt that no standardized achievement test developed to 
date was sensitive enough to be the bottom line of evaluation, 
they willingly offered recommendations on how to increase en- 
gaged learning time. 
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BEST COPY AVAILABLE 



1 

14 

They suggested: 

• Getting students to worK independently and to inter- 
act collaboratively 

• Defining "task" in terms of clear attainable goals 

• Assessing the performance of students and teachers in 
relation to the achievement of the tasks * 

• Installing a managecaent information system that 
helps the teacher keep track of the child's skill 
level and indicates the next appropriate task 

• Reducing interruptions of learning time that occtjr 
when children or teachers are required to leave clasti 

• Matching instructional strategies to learning objec- 
tives and children's learning styles 

• Providing varied approaches, adequate practice-time 
and multiple opportunities for learning and succeed- 
ing 

• Designing a program and appropriate curriculum, so 
that tasks would be congenial to the learning styles 
and the cultural interests of the children 

• Getting aides to relieve the teachers of clerical 
work. 

• Enlisting the parents to help children study at home 

• Enlisting the students in activities that would make 
them more self-determined learners 

• Seeing to it that teachers conveyed their belief that 
children could learn to the children 

David Berliner himself indicated that a «lnqle class- 
room emphasis on items covered by the tests wouxd be unfortu- 
nate: 

Most of the enriching, motiveting, stimulating talk that goes 
on in a class is effort that will not be reflected on the tests. 
Activities that are geared to stimulate students to begin to pace 
themselves, to build on their own sense of responsibility towards 
the outcomes (which wilt serve them later i i U fc) wi 1 1 not be d i- 
recti y reflectc.d In the tests. 
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And he warned that Ergaged Learning Time was no 
panacea. "At the beginning, when a subject is being intro- 
duced, there is a very strong relation of quality time spent 
and a high success rate to outcome, but later the relation 
drops," explained Berliner, resulting in a problem of dimin- 
ishing returns. "Nor are the results of ELT as predictable 
when working with high ability students." 

Testing and Time- On- Task 

Although increasing time on task has not been a concern 
of test meikers, two presenters at the Pittsburgh conference 
described how tests could be designed to be useful to teach- 
ers and children for just this purpose. 

Walter Haney from the Huron Institute proposed that 
tests be developed from which children and teachers could 
learn and he suggested that a reasonable place to 
begin would be with theories of learning (such as Benjamin 
Bloom' 5 theory of mastery learning). If tests were to 
be designed as learning instrimients , they might provide: 

1. Cues that could be altered or adapted to 

the needs of particular 

learners, i.e., written cues for some 

students, oral cues for others 

2. Opportunities for active participation 
and practice with differences in the 
amount of practice or participation depend- 
ing on the individual learning style 

and needs of students 

3. Reinforcers which would be adapted to 
the particular learner (since what is 
a reward for one child may not be for 
another) 

4. Quick and corrective feedback for students 



An important part of increasing engaged learning 
time is hitting the mark with teaching strategies, and 
some of Haney 's suggestions show that he not only had 
thought out how to zero in to the child's needs and in 
terests through tests, but also how to give that all 
important quick feedback. Specifically, he suggested 
that tests should; 
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be available in alternative modes of 
presentation 



• be presented in "game forms" familiar 
to the children who are taking them, 
e.g., word-wizard games, rather than 
vocabulary tests 

• be self-scoring or scorable by individual 
test-takers 

• be of variable length (not all children 
can sit an hoar) 

• provide results not only on whether the 
answers are right or wrong, but on the 
pattern of the errors or sources of corrective 
instruction so teachers can design lessons 

to correct errors in xinder standing 

Other ways tests can be used to increase productive 
learning were proposed by Dalton Miller- Jones, a University 
of Massachusetts professor, who presented at the Pittsburgh 
conference. According to him, teachers can fruitfully 
look beyond the technical errors of children (which provide 
guidance as to what needs review and practice) to the 
ways children process information. 

Pointing to the research evidence, he hypothesized 
that many minority children have divergent test- taking 
strategies. It would appear that some minority children 
also process material differently and develop unique, 
often ineffective learning tactics. "Now that we suspect 
they think differently — we should learn how they think 
differently — so we can help these children better accommo- 
date to the demands of schools, ** said ^Miller Jones. If 
the clinician working with children from culturally differ- 
ent backgrounds can understand the children's logic and 
their cognitive problem- solving patterns, then lessons 
could be better matched to the needs of these children , 
and real progress in correcting patterns of school failure 
could be made. 
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All in all, participants at the Portland and Phila- 
delphia program- planning conferences plus those at the other 
conferences neitJier agreed €iinong themselves on desirable 
characteristics for the nev programs nor about the proposed 
NIE foci of systemic change and tine- on- task, but they presented 
numerous insights in these and other areas about effective 
instruction. As a result of this input, NIE discarded the 
systemic change focus as unmanageable but retained the em- 
V phasis upon tiiae>on-task. 



I 

Dm Principal and Peer Support 



One of -the greatest frustrations, the teachers' lack 
of fidelity to the old FT models^ was alluded to frequently 
at the Portland and Philadelphia conferences , but only a 
few of the participants actually analyzed the factors lead- 
ing to the situation and made suggestions for how it could 
be corrected. 

# 

Researchers and evaluators who had been stymied by 
f the lack of discernible implementation upon wUich to base 

their reports tended to distribute the responsibility bet- 
ween the model designers, who they felt may have insufficient- 
ly articulated and casBm2ni«.ated the behavior teachers were 
to engage in, and the teachers, who even when "well prepared" 
were slip-shod about following guidelines laid out for them. 

Sponsors were more likely to informally blame the 
teachers. Even those whose programs prc^uced notable results 
obliquely disparaged teachers by claiming that their methods 
were "teacher proof". Those model-sponsors of old FT 
programs that failed to produce dramatic results, complained 
that their model's true effectiveness was never properly test- 
ed because so many of the teachers they trained, ionored the 
guidelines and returned to their inadequate but familiar 
style of teaching as soon as the sponsor- trainers left the 
school. 

There were, of course, teachers, administrators and 
acadonics who defended teachers. But no speaker matched 
Robert Stahl in his unequivocal respect for the professional 
in the classroom. Stahl, the manager of the Instructional 
and Professional Development program of the California Teach- 
ers Association, felt that committed teachers had to be recruit- 
ed to make a program successful, and he suggested ways to 
"court" them. 
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■Tell them what kind of assistance Uiey will get. Tell 
them what kind of staff training will be involved and how 
raich time it will take. Tell them what rr.w skills they must 
master and if those skills will serve them after the project 
is over . . . Treat them with respect . . . Get them to 
volunteer for the new program. Unless an enthusiastic 
explanation of i^at the project can do for a teacher is given, 
expect reluctance f" he cautioned. 

After the best teachers have been selected ax«i 
insenrice training begins, programs should develop methods 
S Sintain teachers' coomiitment. The work-group concerned 
%rith building internal support systems, advised that program 
administrators: 

• Give participants "a piece of the action" by 
involving them in planning the inservice program. 

• Bring in an evaluator to help define the project, 
to work with the teachers to decide on the scope 
and size of the escperiment, and to design the 
formative and sunmative evaluations* 

• Consult with participants about scheduling staff 
development programs and be prepared to offer 
training during the instructional day or to pay 
teachers for their overtime. 

• Invite professional collegial interaction 
among participants. 

• Focus on what teachers do, not what they are . 

• Build the program around what is already working 
effectively in the school. 

• Provide a plainly written explicit description 
of the program goals and practices. 

• Make the staff development program responsive to 
a school needs'assessment in which the teachers 
participate. Try for staff consensus. It's the 
best assurance that the program will be implemented. 

• Keep meetings short and distribute a written 
agenda in advance. 
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• Inclxide both theory and practical ideas for, 
the classrooa. 

• Spend some time on the technology of the 
instructional model and the management of 
class tiice. 

• Keep the expectations and capabilities of the 
staff in mind when training them. 

• Find ways to get teachers' honest reactions to 
the training as it proceeds. 

• Give teachers the tools, the time, and the 
training to enable them to observe and diagnose 
children and make informed decisions about their 
learning needs. 

• Train substitute teachers along with the regular 



• Recognize that programs will vary from site to 



• Advise teachers how they cam get help frcim outside 
resources such as teacher education associations, 
teacher resource centers, netwrks for tsachers, 
teacher exchanges, and tflrained volunteers. 



University of Colorado, Boulder, argued that statistical evi- 
dence shows that innovative educational treatments have little 
appreciable effect and research evidence bears this out. But 
teachers rarely study the "evidence.* Rather they embrace or 
reject prograns on the basis of their understandings, beliefs, 
and wishes 7- reasons which make more sense to Glass than 
"the arbitrary and illegitimate authority of evaluators.* 
Believing that researchers who want to help teachers should 
emphasize qualitative description, he would have thieir 
reports include: 



• Some coherent, detailed portrayals of life in 
school for pupils, teachers and parents as it is 
colored and shaped by allegiance to a particular 
instructional model 

• Some portrayals by disinterested, expert ethnographers 
with at least two years on-site for data collection 



staff. 



site. 




researcher from the 
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• Some portrayals focused on a broad range of 
concerns including the model's philosophy, its 
history (since its future must be projected) , 
techniques, financial and psychic costs, side- 
effects and after-effects, roles it requires 
people to play, its potent >r a f&vorable 
evolution f and the like 

"Our evaluations should not attempt to discover the 
one or two right programs for all children and get everyone 
to follow the prescription," concluded Glass. "We need eval- 
uations that will help school people make informed choices 
based on their goals, their philosophies and the nature 
of their districts." 

Midway between Stahl and Glass' position that most teachers 
conscientiously <*oose to adopt, adapt or ignore innovations and 
the contention of the disappointed model sponsors that irany 
teachers mindlessly subvert programs, is the perception of 
Judith Little;, froa the Center for Action Research- she claims that 
even the best teachers yith aood intentions often cannot faith- 
fully adhere to a system they believe in if they are not 
monitored frequently— their resolve is just not enough when 
classroom exigencies come along. Little attributes better 
teaching to increased staff collaboration 7- wl^at she calls 
"the norm of collegiality" which is very difficult to build 
but well worth it. In her presentation at the Washington 
Conference, Little described how teachers in more effective 
schools behave differently from those in less effective ones. She 
reported that the teachers spoke to each other about teaching 
in precise terms that communicated subtleties about their 
aims and techniaues. They observed one another frequently 
and having learned to separate practice from personality were 
able to give and take criticism. They were willing to articu- 
late their objectives and get feedback from peer observers 
on how well thev were meeting those objectives. The planned 
and prepared lessons in teams and learned from teaumates how 
their lessons could be clearer and more interesting. Accord- 
inq to Little all of this takes trust, intelligence . tact, 
honesty and a willincness to change or modify classroom 
techniques — but it results in more effective teaching. 

Little also oointed out that successful teachers were 
usually aided bv strong principals, who were educational 
leaders rather than nerely adept administrators, a point 
confirmed by :;iS*s Virginia Koehler and Ursula Plnern 
citing evidence from effective- schools research. 
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Since, as Plnero pointed out, most principals are trained 
as administrators and selected for their administrative abil- 
ity, most do not have the skills needed to be dynamic educa- 
tional leaders. 

The project directors attending the DC meeting in pre- 
paration for turning educational insights into workable proj- 
ects wer<3 well aware that they might have to do as many s?vvy 
dist-zict administrators now do — make %^rkshops available to 
pxmcipals as well as teachers if they were to radically im- 
prove thdir sch(X>ls. 

Principals need to be well inform^ about instructional 
methods, have better supervisory techniques and take a more 
assertive role in the selection and training of new teachers, 
according to the speakers. 

They must be prepared to make regul€u: visits to class- 
rooms and follow them by meeting with staff to suggest im- 
provements — in a non-threatening way. They xttust encourage 
experimentation and make provision for teachers to plan to- 
gether rather than discouraging teacher initiatives. 

They must communicate their faith in students and staff 
by setting high expectations and by officially recognizing 
effectiveness, providing a safe and orderly school, a system 
for monitoring and assessing pupil performance, and a school- 
wide emphasis on basic skills instruction. 

The imperatives of Little, Ko^hler and PiJJero reinforced 
the advice given earlier by Portland and Philadelphia conferees, 
that efforts to strengthen the instructional process address: 

a. The attitudes and expectations of the staff, 
with special attention to the involvement of 
the principal 

b. The organizational structure and procedures 
including student/staff assignments, time 
allotments, and role responsibilities 

• 

c. The system of incentives and rewards that will 
be used to encourage high-level performance 

d. It and how teacher associations will be included 
in the negotiations 

e. If and how the local Board's commitment to the 
new model has been assessed 



Bow to intxoduce new laaterlals, new structures, 
new teacher behaviors and new vtilues 

Bow to work with the staff so these new structsares, 
behaviors, and values and "internalized" 

A plan that is responsive to a demonstrated 
local need. 

A i^ll-thought-out delivery system which Includes 
the training o£ parents 

An effective management system 

A plan for involving spb'hsors or outside advisors 
can help maintain the quality of the curricu- 
lum, train staff, and work with teachers and 
parents without causing conflict. 
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^SEARCH AND EVALUATION 



Evaluations that do not dig deeply enough to reveal 
what has really happened are worse than useless. They may 
damage the educators and the children involved. For the 
educators, they deny hcurd-won advances. For the children, 
they obscure benefits that could, in turn, help educators 
elsewhere to help their pupils. 

For example, a*- roughly the same time as the BTES 
researchers were compfuring the effects of teacher management 
skills on students, other NIE funded researchers were challeng- 
ing an educational shibboleth generated by the Coleman 
Report on the Equality of Educational Opportanity — that 
success and failure in schools was primarily a matter of social 
claos and family background. Public interpretation of the 
Coleman evaluation and research results — that schools and 
teachers could make very little difference constituted 
a "formidable obstacle to advancing [educational] equity" 
according to Ronald Edmonds, a key researcher and dissemin- 
ator of the Effective Schools Study that contradicted coleman's 
findings. According to Edmonds, Koehler, and. others who 
have spoken about the effects of the Coleman study, the 
belief that their work would not have much of an effect dis- 
couraged school people from trying to iropr eve their techniques. 

In contrast, the Effective Schools Studies have encouraged 
school people to find ways to improve their teaching and managr- 
ing techniques. 

How could two studies have produced such different findings? 
One collects budget figures, listed comparable facilities, 
numbers of teachers etc. and ccrapared them with the achievement 
scores. The other, looked carefully at the few high achieving 
schools in the ghetto which challanged the average expectations. 
The Colo&an study lost important results by averaging good 
schools in with the failures, thus masking the real differences 
between effective and ineffective schools - diff^ehces which 
were more important than facilities or budget allocations. 
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Is "It" Being Done? Can It Be Done Again? 

It is reasonable that no program's results should be 
tabulated or publicized before it has been determined that a 
specific treatment has been administered. Yet, determining 
implementation in educational research that is not necessar- 
ily the noina. 

The probl«n became painfully clear when attempts to 
conscientiously verify the "treatment" of earlier Follow 
Through planned variation models floundered. Treatments were 
insufficiently defined and usually unstable, and there were 
too many variables (such as the styles of the teachers, the 
backgrounds and experiences of the children) and hosts of un- 
predictable social forces (such as strikes, student attrition, 
community tensions, etc.). 

All the major Follow Through evaluators were frustrated 
by the lack of specificity of treatment in earlier FT 
models. They complained of "inconsistency of classroom be- 
havior within the program" (Soar) ; "imprecise treatment defi- 
nitions and descriptions" (Emrick); "a lack of explicit state- 
ments by sponsors of what proportionj of the time-critical var- 
iables would occur in an ideally implemented classroom" 
(Stallings)? and of "inadequate assessment of whether the 
sponsors ' evolving programs were implemented" (Abt) , 1 

Equally frustrated were the FT model-sponsors attending 
the conferences, who expressed themselves on the issue: 



• "The local school district refused ccsnpletely to 
ccOTiply with our guidelines," said one model -sponsor. 
"And even after we complained to the Feds they got 
away with it, because it was a political money grant, 
not one basec on the site's willingness to adhere to 
our guidelines. We learned we had no clout." 

• "Every teacher in that city 'messed up. ' The city 
was famous among all the sponsors. Whatever those 
teachers were doing, it wasn't what we had in mind. 
They just went their own way." 

• "it seemed the further a district was from our home 
base, the less apt they were to be implementing our 
program," said another sponsor. 



Haney, Walter. The Follw Througn Planned Variation Experiment 
Vol. V, The Follow Through Evaluation; A Technical History . F> 157-159. 
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There were exceptions. Sponsors of models with specific 
easily described teacher-behaviors and even a "script" for 
specific academic activities had less troiible getting teachers 
to behave as directed than models that depended upon rela- 
tively vagiie changes of attitude and philosophies. It was 
evident from the national evaluations of the Planned Varia- 
tions that the most prescriptive classroom models were more 
iiniforni across sites and the children averaged higher academic 
achievement scores. But beha\-iorist models were not popular 
with the majority of the educators present at the Portland and 
Philadelphia meetings, and so problems of implementation re- 
mained of great concern. 

Responding to the problems of implementation and repli- 
cation, one Portland work group proposed an entirely new for- 
mat for the next Follow Through research project. It vrould 
begin with a search for exemplary programs, and proceed to a 
aetailed description of the successful programs and the fac- 
tors which might help explain their success. Out of this would 
develop a verifiable theory of effective early primary educa- 
tion, including the necessary adaptations to student differ- 
ences and other system variables. The project would continue 
to a study of implementation and end with a study of the fac- 
tors that make for effective replication. Why start again from 
the beginning? Walt Hathaway, the Director of the evaluation 
program of the Portland Schools, and of the work group, ex- 
plained: 

Without a well-defined accurate description of the rrTodel 
vyti dor't know what happened or it happened. Without a look 
at the context of where it happened and hw it happened, we don't 
i^now what v „nt into making it happen. 

We must know the "where, how and v/hat " to know if It can be 
transferred from one site to another or if it can be generalized 
at all. So after we understand the model, we have to take a look 
at "receptivity" of the receiving system (the management, the 
staff, students, parents, etc.). 

Hathaway called it a "holistic model" of systemic change. 

This group's recommendation that the "ecology" of the 
successful site be thoroughly described was elaborated upon by 
the Austin conferees. They concurred that detailed descrip- 
tions of implementation, including "climate" of the school, the 
school district, and the community, made replication more 
likely. 

The complexity of the problem of implementation was 
graphically depicted in Austin by Michael Fullen from the 
Ontario Institute for Studies in Education: 
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PROCESS OF IMFLEMEKTATION 
(Michael FulleSp Ontario Institute for Studies in Education) 



FACTORS AFFECTING 
IMPLEMEKTATIOK 



THE CHANGE 



A nev 
program 

or 
model 



A. Characteristics of the 
Change 

1. Need 

2. Clarity /CoiopleKlty 

3. ?laterials Quality 

B • I&pl esen t a t ion 

4. Adoption 

5. Staff developtDsnt 

6. Tine-line 

7 « Internal/External 

C« District Factors 

8. History 

9. Adninistration 

10. Parent s/C<»munity 

D. School Factors 
11^ Principal 
12. Teachers 

E. Extraneous Factors 
13* tnanticipated 

events 



IMPLEMENTATION 



OUTCOMES 



Changes in: 

1. Materials 

2. Structure 
3« Teaching 
4. Beliefs 

(involving 
teachers* aides, 
and parents in 
instruction) 



Achieve*** 
went At- 
titudes 



"Icplenentaticn a&ans changing practice, and its complexity Is suggested 
by the dlagrais a*::o\^e« The laind will be excused for boggling at the prob**- 
lee, since it Includes all of the above issues and nK^re: measuring all the 
inputs* ireasuring the various aspects of implesientationy testing for a vari^ 
ety of outcraest interrelating all three sets in order to compare very dif- 
ferent FT codels and going beyond that to compare them vlth non-FT class^ 
roons. My ovn approach vpuld be « « « (1) to develop common measures of the 
inputs* (2) to explore some comoson implementation iwasures* but also rely on 
custom measures of icplenentation unique to each model and (3) to use same 
coc3on outcome measures* but also rely on some custom measures unique co 
each model vhich vill contribute to broadening the range of outcomes meas-*^ 
ured.** 
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Vhat is the Teacher Doing? 

Also struggling with the problem of in^leuentation, Chad 
Ellett from the University of Georgia proposed a plan (de- 
signed with researcher Margaret Wang from the University of 
Pittsborghl vhlch would define the critical dimensions'* and 
provide "scaled descriptors" for each aspect of a program 
under study — to enable an evaluator to decide whether a 
teacher was actually implementing a program and how. 

With the observation instruments proposed by Ellett and 
Wang (or coo^wrable ones) , an observer could determine if the 
teachers* behavior conforms to the ideal teacher-behavior 
for that instructional model. Such instruments wc»dd also help 
administrators assess the effectiveness of suqpport systems for 
teachers. By charting classroom behavior before and after the 
teacher participates in workshops and conferences, it would be 
possible to determine if change has occurred. 

But change in teacher behavior is generally only an 
interim goal. The ultimate goal is change in the children's 
classroom behavior and academic achievement. Clearly we must 
know not only wLat the teacher is doing but what the children 
are doing. We assume the teachers* behavior will affect the 
group's behavior. We also know that all children do not 
respond to a teacher the same way, a point driven home by 
Technical Advisors Gary Borich of the University of Texas and 
Robert Floden of the University of Michigan in their expert 
presentations at the Washington meeting. 



What Ar 2 the Children Doing? 

Looking at the children isn't as simple as it sounds. 
Both Floden and Borich showed why choosing an approach ccmipels 
evaluator s to make "trade-offs." Such a choice must be 
made when deciding whether to observe a small number of 
youngsters who represent different learning types or the 
whole class. 

watching a few select ixsdividuals 1ms certain advantages: 
it enables the observer to make refined judgments which can 
help a teacher predict what type of children the program works 
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best for and in what circumstances. The obsexrver will know 
when Alice is inclined to daydream and what helps her focus; 
what subject is easy for Charles and whether success energizes 
him; how a lively pace affects Daniel and children like Daniel. 

But It presents problems too. If the focus has been on 
six children, and one sixth of them leaves, the study has 
considerably less validity. 

So, rather than take that chance, the observer might 
choose to look at every other child in a classroom, miile the 
observation will not be sensitive to how specific instruction- 
al techniques affect certain types of children, the broader 
base of data gathered can be used to determine the overall 
effects of a program. The observer of 28 children will not 
know if Billy's iK>dding behavior is merely learned "nodding 
behavior" or a sign of serious concentration, but will have a 
sense of how the class, as a whole, is responding. 

The choice about wfacm to watch is relatively simple, 
compared to choosing how , when , and how much to watch, accord- 
ing to Floden and Borich. 

If an observer charts (or photographs) what each child 
is doing every 15 minutes over the course of one hcur, and, 
after 14 minutes of concentrated %#ork, Carol notices her un- 
tied shoelace, is that a fair measurement of her general fo- 
cus? If after a chaotic transition, the teacher calls the 
group to order at the nManent it Is time to record the observa- 
tion, is that a gauge of their general involvement? 

Should moments that a child is looking out the window 
gathering thoughts be recorded as a "miss** in fccus when it 
results in intense witing for 10 minutes after that? If that 
is actually engagement, would it be so seen by an observer who 
was charting all 30 children in a room? 

What Instrument for recording shall be used? How many 
items should be on the talley sheet? How inferential should 
the items be? Should an observer visit 10 consecutive days 
for 2 or 3 hours at a time to see how children function day 
after day or space visits on a weekly, monthly or bi-monthly 
basis? According to Floden, results are influenced by the day 
of the week, the time of year, the outdoor temperatiire ^ the 
amount of rain, etc. How should a program observed in January 
be compared with one observed in iMay? 

To conclude, while any reliable and valid research 
study requires ac irate assessment of implementation, such 
an assessment (pe iiicularly when a large group of children 
is being studied) is difficult. While it is certainly 
easier to ignore individual differences and treat children 
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as a group — strong positive and negative responses to 
a teacher's management style can be missed if responses are 
averaged out- By washing out the real effects of the program 
on particular types of students, the research could incorrectly 
assume that the program has little or no impact, when in fact 
it may work superbly for some children and be dreadful for others. 

A Grab-Bag of More Problems 

Evaluation problems only begin with the questions, are 
teachers implementing with suffTHi^nt fidelity and are chil- 
pJ^L^,^^ J"^^^" <^onsistent way. Researchers at the 
Pitt^urgh, Austin and Washington conferences posed other 
equally seminal questions, such as: 

Hov stable is the student population? 
How appropriate is the content and format of the test 
instrument? 

tLe program match the children's educational needs? 
What IS the result of the program over time? 

• Robert Floden pointed out that in some communities the 
target population turns over at a rate of 100% if the 
schoolchildren who begin the program have mov4d by the 
end of the school year and children from other schools 
have replaced them, how should an evaluator treat their 
end-of-the«-year scores? What percentage of the year 
in-class constitutes sufficient exposure to a program 
to warrant score-inclusions when averaging results? 

• Children can be "disoriented" by changes in teaching 
techniques, teacher expectations and classroom manage- 
ment according to Leigh Bur stein from UCIA. His in- 
sight raises such questions as should the effects of 
change be taken into consideration when monitoring test 
results? Should we treat the test-scores of children 
who have been in a program over an extended period 
separately from those who are "disoriented" because 
they are new to the program? 

• Language proficiency or the lack of it distorts the 
achievement results of bilingual children, according, to 
Ernest Bernal of Creative Enterprises, Austin, Texas. 
As he pointed out, a program will appear spectacularly 
successful when non-English-reading children finally 
learn to decode the tests. And, alternately, it can 
appear to be dismal failure if the children do not ac- 
quire the reading skills needed to take the tests. 
Bernal 's presentation raises the qxsestion, should' 
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non-English-proficient children be given different tests 
to make their scores reflect their real accompli sbnents? 
If in^ssible, should their test results he pulled from 
the groups and handled differently? 

Learning strategies "invented" by some childr^ in 
their early priisary years are found to be dysfunctional 
vtien additional learning demands are made, according to 
Dalton Miller-Jones from the University of Massachusetts 
at Amherst. At that point, test scores may drop pre- 
cipitously. Jones finds this is particularly true of 
"minority children." Based on Jones' findings, one 
wonders if a superior educational approach might not be 
underrated, if those who expect continued progress don't 
take into account the need to fashion new learning 
strategies at this critical juncture. 

Standardized tests are frequently not an acciurate 
gauge of a program's effectiveness according to Floden 
and Richard Jaeger from the University of North Caro- 
lina. Both studied the literature and found that it is 
likely that no ji^re than 40% of what is taught in class 
will be reflected on the ataz^ardized achievement tests 
(the range is 0% to 60%) . If more than half the test 
questions are inappropriate (even when dealing with a 
traditional arithmetic program) , why not give tests 
which do measure what is hapi^nlng in the classrooms? 
Because, according to them, locally designed tests are 
difficult to con^are to nationally normed tests. It 
would be almost impossible to ccanpare alternative in- 
structional programs. Could we circumvent the problem 
by judging the amount of change in the children, rather 
than their absolute scoresT" Not easily, because as 
David Berliner and-otiiers have pointed out, rates of 
academic accomplishment depend on where the children 
begin. 

SOTie programs appear to be enormously successful at 
teaching decoding and ccniputation but, according to 
critics like Lorraine Smithberg from Bank Street in New 
York, they do this by sacrificing understanding which 
must be the basis for continued cognitive growth. The 
designers of primary programs which emphasize hands-on 
experience, discussion, dramatic and block play, group 
trips and personal writing (e.g., the Bank Street method 
and methods based on Piaget's work) assert that the 
real test of a program's effectiveness lies in the 
capacity of the children to demonstrate as they mature. 
Are they developing analytical skills? Can they take 
control of their learning and their lives? 
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•Robert Egbert, the former director of the Follow Through 
program, advised that children be followed through the years 
to determine a program's long-term effect. He proposed that 
in addition to keeping tabs on their academic achievement 
through the years, records be kept on how many children must 
be placed in remedial classes or held back, how many drop out, 
get involved with drugs, engage in vandalism or other anti- 
social acts. 
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Hon RESEAPCH STUDIES AFFECT ATOPTION AND ADAPTION 



What do administrators need to know about a program before 
they try to adopt it or adapt it? Most researchers believe 
they n^d to knoiri 

If the program was in^lenented 
Bow it hapi^ed 

IBlat personnel were involved , 

What the children were like coming into the program 

t9hat they were like after being exposed to it 

How they shaped up in the follow-up studies 

The economic, social and ethnic characteristics of the 

conmmity 
The ecology of the school. 

Some researchers « like Jane David of the Bay Area Research 
Group, hoped that in the upcoming round of research, the same 
model would be isplemented in a number of sites with different 
eco-systems so comparisons could be made. 

Although their plans differed, researchers (like David , 
Ellett, FullCT and Hathaway) who reccaraended elaborate imple- 
mentation sti^lies begin with two common beliefs: - one, the way 
implerontation had been ai^oached to date was inadequate, and 
twb, the right data and "better science" (a phrase introduced 
by Yale's Edward Ziegler) would yield information that would 
make replication possible. Even if schemas for getting a grip 
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t^^aZlii^i^ ^^^^ Michael FuUen's, pg.26 j resulted in 
ii^^ complex fonaula, still one could imagine that 
^® might be sufficiently catalogued and 

sorted by data banks to be useful to school policy makers. 

But there were a small number of maverick researchers who 
expressed strong doubts about the wisdom of attempting a 
scientific- analysis of the factors. 

l^tl argued that finding a single set of 

instruments that could apply to many schools was less im- 

K^^/^^i"^ studying what was happening in 

^Lnf school. To her mind, instruments designed to bl 

Kfore''''f^^w^^i^ ^^T'''' ^ site, 

merefore, studies should be content to feature and emphasize 

^t teachers in that particular system did to improve their 

others, likfc Ernest House from the University of Illinois 
and Yvonna Lincoln from the University of Kansas /challenged 
* hard science- in the search for stable truths in 
human Affairs. In fact, they challenged the very idea that 
there were any stable truths to be found ir. human affairs. 

House opened his Pittsburgh presentation with a tale of 
his two children. Exposed to "the same" innovative reading and 
math programs in the same school, one child flourished, the 
other was bored. Why didn't they have similar reactions? The 

^ f f " different learning characteristics and ex- 

pectations. By the time his second child went through school, 
^^LS^'i^i?^ teacher was replaced by a younger, less energetic 

ccanaitted to the program of instruction, 

explained House. 

4r^4^^^^ message was clear. If two children from the same 
ramiiy, attending the same school, could have such different 
reactions, would even a thorough analysis of a program be of 
practical use to disparate districts? since local factors 
(staff, children, community) all affect a program, it seemed 
unlikely that a universal prescription for success could be 
formulated even out of superb data. 

. The lively discussion that followed House's presentation 
in Pittsburgh (and a similar if milder reaction following 
Yvonna Lincoln's presentation in Washington) focused attention 
on the twp contesting positions in the pantheon of the re- 
search community. One group trusted the scientific method to 
find an answer to any question — if only the right question 
and the right data could be marshalled. The other group 
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trus-ted people in a schooX system to make intelll9eiit de- 
cisions based on their experience and aided by an anecdotal 
history of previous experiments. 

At the Fdrtland and Philadelphia conferences it was clear 
that although many humanistic model-makers were at odds with 
their more detexminist counterparts in matters of prograoa 
development, they shared the belief that "their* program could 
serve all the children best, if only done "right; " 

In ccaitrast, seme outspoken presenters took issue with 
the search for any "right" way: 

• Eugene Glass from the University of Colorado, Boulder, 
doubted that any formula program ever did much good. 

• Thomas McNamara frooii the Philadelphia Public schools 
thought the secret lay in putting new methods "on 
trial" ^ere a local synthe&iis of the points made by 
advocates and dissenters would result. 

• Susan Loucks from the Network, Inc., Andover, MA, pro- 
posed that innovation requires a process of . absorption 
which starts as curiosity, moves to fidelity, on to fin^ 
tuning, and finally to adaption. Her Concerns Based 
Adoption Model, while it doesn't specifically challenge 
the search for scientific truth, does suggest that the 
secret of successful implementation lies in the wisdom 
and instincts of the teachers — not in research data. 

The unsettling challenge of Bouse, Lincoln and Glass 
vas a formidable one. fftiile their mere traditionally or- 
iented research' peers were not wcm over by their arguments, i 
their view was accorded considerable respect. One observer/ 
was startled that researchers faithfully "married" to the ' 
scientific method put up little resistance to the radical 
critiques of it. 

The Pittsburgh critics of the scientific approach were not 
the only group to take issue with traditional research. Prac- 
titioners in Portland also spoke out against "the collection of 
massive amounts of data which has in the past put an unuaaal teden on tbe 
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school i^ystejB and on teachers." They suggested that princi- 
pals protect teachers from evaluation efforts that "have 
nothing to do with school reality so teachers could be free to 
concentrate their best effc. ts on teaching." Eva Baker at a 
later meeting also took issue with the "necessity of extended 
research that did not directly help teachers and schools do 
their jobs better." 



Spreading the Word Effectively 

In the eild, while the amount of time and energy which 
shoul,d be committed to implementation studies and site anal- 
ysis — the content of reports to the field — remained moot, 
the need for effective comaunication was incontrovertible. 
Everyone easily agreed Cwith work-grwaps on communication) 
that reports should be tailored to meet the needs of each 
audience. Specifically, they suggested: 

• When addressing the general public, focus on outcomes 
and use simple percentages rather than con^jlex statis- 
tics. Make the report as clear as a Reader's Digest 
piece. 

• When addressing the educators, parents and community 
decision-makers, be specific about student outcomes, 
cost-effectiveness, and how the "target" populations 
will be selected. Tell them what teachers will be ex- 
pected to do differently, demonstrate the programs' 
operation and materials and, above all, pitch the talk 
at the language level of' the group. Avoid jargon. 

• When addressing district administrators, be specific 
about expected outcomes and present in detail the addi- 
tional requirements or impositions of the program. 
Demonstrate the materials and present the research 
results in nontechnical visual way. 

• When addressing state Departments of Education be 
specific about how the program will or will not be in 
compliance with state regulations. Tell them what kinds 
of dissemination the district will engage in, and, if 
possible, get a district superintendent to do the pre- 
senting. 

• When addressing federal administrators, be specific about 
the cost-effectiveness and how the program is to be mon- 
itored at the site level. 

When possible, have an administrator address administra- 
tors, a teacher address teachers, «etc. 
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THE NEW FQLLG^-TBROUGH PILOT PROJECTS 



Guidelines foe the Prolects 

The HIE' 8 Request for Proposal (RFP) issued on June 10, 
1981 si^cified that proposed projects serve sl^udents in grades 
K-4 during the 1982-83, 83>84, 84-85 school y^ars. Activities 
prior to the 82-83 s^iool^ year were to be planning, base line 
data oolleetion and other preliminary tasks. The RFP asked 
that the projects generate information to help other school 
systems consider whether to adopt all or part of the program, 
hy providing the following kinds of data: 

^ • h description of the project including the planning 
process, the implementation process and its major 
characteristics 

9 Student achievement, in reading and mathematics 

• Student and teacher use of instructional time 

• Attendance 
e Costs 

e Student plac^nent, promotion and retention 

• Parent, student and staff opinions about the project/ 
school. It was required that each school district 
provide two schools (small districts' could join with 
another district to present a prcx>osal) and all child- 
ren in grades K-4 et each school be involved in the project. 
The schools were to be selected because at least 5C per- 
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of the children had Head Start or similar preschool 
experiences. 

Recognizing the challenge of keeping costs for such 
a project realistic, in view of federal and local cut- 
backs, and to assure that the new projects received a fair 
test of their effectiveness, the NIE maiwiated that students 
involved in the projects should not receive conpensatory 
services funded by other st;tte or federally supported educ- 
ation prograas. . 



In the summer of 1981, after the Institute's RPP stim- 
ulated nationwide response, contracts were awarded to: 

The Cotopaxi (Colorado) Ccmsolidated Schools for 
a joint project with Westcliffe; 

The Napa County (California) Office of Education for 
a joint project with Napa Valley and Vacaville: 

The Oakland (California unified School District; 
The Detroit Public Schools. 

The NIE furnished each district in the amounts below 
for the costs of developing inplenienting and executing 
its pilot project over a five year period: 

Detroit $638,037 

Napa $404,531 

Oakland $666,072 

Cotopaxi §240,311 



The average yearly per child cost over the districts* 
usual allotment is $153 — an amount below the average per 
chxld expenditure in Title I of the Elementary and Secondary 
Education Act, which is the nation's largest compensatory 
education program, it is also well below the per child costs 
of exxsting Follow Through programs. 

The Institute also arranged to have six experts in staff 
relations, program development and evaluation on-call to the 
districts to help them plan their projects. For four months 
these Technical Advisors would be available without cost to 
the districts for occasional visits and phone consultations. 
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A. The Detroit (Michigan ) Project 

Searching far cost-effective inservice experiences which 
would lead to more effective teaching « the Detroit schools pro- 
posed four levels (to be used singly or in combination) — .all 
of i^ch rely on peer support: 

• ^O'fle^C' of Theories and Practice s Teachers will be 
guided to materials and articles in the school library 
related to their areas of difficulty. This could range 
from an introduction to the theory and practice of 
Mastery Learning or Direct Instruction # to a review of 
classroom oanageiaent techniques such as a focus on 
classroom discipline or the physical set-up of a 
classroon • 

m Observation to Correct Problanatic Behaviors ; Teachers 
will c^serve others who are identified as highly 
skilled. In addition teachers may be offered the 
chance to view themselves on video tape so they can 
critically assess their otm teaching progress. 

• Practice in Simulated Conditions ; Teachers will be 
encouraged to try out specific behaviors with small 
groups of students tinder circumstances which do not 
require the management of an entire group. 

• Coaching s A demonstration teacher will be available to 
coach teachers on a one-to-one basis. 

At the heart of the Detroit plan is a Mirrors and Monitors 
process which would be used to assess individual teacher's ini- 
tial needs as well as needs that remain after each level. 

• All teachers in the program, working in groups of tWo to 
five, will observe one another and meet to discuss: 

a. Their major instructional goals 

b. What they are learning from trying new teaching 
techniques, from w(^kshops ai^ from readings 

c. The results of observing one another teach (how 
much time their partners are on-task and of f -task) o 



Sheldon Sofer, project director, indicated that the class- 
roc^ observations proposed in the Mirrors and Monitors plan 
initially upset many teachers. It took tijm to convince them 
that the information would be kept confidential and would not 
be used in any foxmal teacher evaluations. Only three people 
will see the classroom observations — the teacher being ob- 
served, the observer (teacher observing) and the data collec- 
tor. 
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Detroit's Evaluation Plans 

scientific and naturalistic inquiry methods vill be 
us«^ to ii^ove inservice training and to document the pro- 
SSive^?f^?S«^*^^^ y^^^ provide quantitative and qual- 
co^ol ^S^^S" concerning the nature, effectiveness Ld 
cost of each of the four types of teacher training used. 

Questions to be addressed are: 

• Do teachers increase their on-task behaviors? 

• Do teachers decrease their off-task behaviors? 

• Do students increase their time-on-task? 

• ?w*^i5f^'^*' do students find the activities in which 
they are engaged? 

• Are students engaged in low error-rate activities? 

• Is the time allocated in reading and mathematics spent 
on outcomes which are valued and addressed in the out- 
come measures? 

• What is the relationship between student time spent 
learning particular topics and student learning of 
those topics? 

• Do project students make significant gains on the 
achievement measures? 

• students gain more on the achievement measures 
than students at the -cos^arison" schools? 

T*.^hn?^«^^?Ly^^^ suggestions made by Yvonna Lincoln, one of the 
Technical Advisors provided to the districts, Detroit has en- 
gaged an Oral Historian to (a) document project activities (in- 
cluding meetings and training sessions), interview project 
participants and others felt to have a stake in the effort; and 
{t>) to improve the interventions as the project unfolds. 

^^^o*-!*^^® central purpose of this historian's work will be the 
creation of a real history, a written narrative," according to 
Lincoln, CoBnwntlng further on the purpose she explains: 

"The end product should demonstrate how it was done, 
given this particular social context, and should assist 
both project directors here and school districts elsewhere " 
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Anot^sr sugges-tlon made by Lincoln, who favors naturalistic 
inquiry, was that Detroit use the library checkout systesa 
to provide an unobtrusive measure of the effectiveness of the 
workshops to stinolate further interest. 

In addition to keeping a check on who takes out the mate- 
rials, the library will collect cooanents by teachers who have 
already used the file ai^ make them available to others. 



Achievement Data 

Detroit plans to use the Metropolitan Achievement Test 
(MAT) which they feel "is a good match for the Detroit Curric- 
ulum." But to be more precise about the effects of their pro- 
gram, they also will select subsets of items from the MAT, and 
relate their outccnies to the time spent on these topics in 
class by "engaged" students. 

Teacher- Focused Instrun^nts 

Teachers on- and off- task behavior will be measured using 
two instruments: the Classroom Snapshot Form developed by Jane 
Stallings, and a Detroit-developed coding f om which lists on- 
task and off -task teaching behaviors. Every two minutes dur- 
ing a 50 -minute math or reading class, coders will record what 
they have observed during the previous two minutes. (The same 
form will be used by teacheu.s as part of the Mirrors and Moni- 
tors intervention.) The snapshop instrimient will be adminis- 
tered twice a year> the Detroit-developed form twice a month. 

Student-Focused Instruments 

Four types of data will be collected to monitor student 
behaviors student on- and off -task behavior, student attend- 
ance, error rate on seat- work and timt allocated to the study 
of selected reading and mathematics topics. 

The Detroit seating chart used for on- and off -task be- 
havior will be completed every two minutes during the observa- 
tion period. In addition to coding individual students, the 
form will also provide space for the coder to indicate the s\»b- 
ject (reading or math} and the topic of the lesson observed. 
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Analysis of Findings 



Acknowledging that achievement is likely to be influenced 
by how prepared students are irtiea they enter the program, De- 
troit will group tests for analysis by students* entering level 
Chigh, medium and low) . They expect that students who enter 
low will make greater mastery gains. 



B- The Cotopaxi (Colorado ? Project 

School systems of fewer than 300 students from kinder- 
garten through high school make up one third of the nation's 
schools. Yet while more than one million students in the 
United States are educated in systems of this size, most ex- 
perimental programs have neither been designed to meet the 
needs of these childrex^ nor tested in -these schools. 

Cotopaxi, with 225 students in grades K-12, had to ccmi- 
bine with Westclif f e,with a total of 300, to amass a test 
group sufficient to qualify for this contract (two classes per 
grade) . 

Rising costs (particularly for transportation) in commu- 
nities where 70 to 90 percent of the students are bussed to 
school every day, are cutting into the funds available for in- 
struction. Sensitive to the need of the communities to save 
one -fifth of their transportation, heating, electricity and 
maintenance costs, the Colorado State Board of Education of- 
fered the option of a four- day week to rural school systems 
like Cotopaxi and Westclif fe in 1980. 

Cotopaxi jumped at the offer. Because they vere already 
losing classroom time to sports events (regularly scheduled for 
Fridays) , they saw the four day week as a way to increase 
academic teaching tine and save money. 

Two primary goals drive their effort in the new FT project: 

1. To improve student achievement 

2. To collect and analyze the necessary data to determine 
the viability of the four day school week.. 

To achieve their goals, they have set the following ob- 
jectives: 
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• To reduce and simplify administxative tasks of teachers 
by streamlining and consolidating forms, etc. ^ so 
teachers can focus on instruction 

• To eliminate %msted time in classrooms by training 
teachers in the mechanics of classroom managen^nt, 
i.e., smoother transitions, more effective discipline 
techniques 

• To increase the effectiveness of the allocated academic 
time by defining i^t it is that the children should be 
learning. The developnent of a K-6 curriculum guide 
vdiich sets grade tasks and criterion- referenced tests 
which help teachers assess their classes' progress are 
two of the first orders of business. 

To help teachers make the m>at of their contact-time with 
the children, the Colorado project will conduct the following 
activities: 

1. A consultant employed as a data collector will visit 
the classroos^s to observe and record Engaged Tin^e. 
These observations will provide an insight into the 
causes of low engagement in the target schools, such 
as lax discipline practices, lack of motivation, un^; 
tidy transitions, poor organization of material, etc. 
At least 10 day-long workshop sessions for teachers and 
administrators run by outside trainers will focus on 
the most needed correctives. 

2. After each inservice training session the data collector 
will observe the beachers in action. The notes from 
thes^ observations will be passed along to the principals 
and teachers and form the basis of the one-on-one 
conferences between principals and teachers. Together 
they will discuss the ways the teachers might improve 
their approaches. 

3. Teachers will be given the opportunity to visit 
other classes throughout the area. 

David Trujillo, Superintendent of the Cotopaxi schools 
noted that Cotopaxi "has an advantage over most school systems 
that must deal with union rules and a five day week, because 
teacher attei^ance at staff development activities can be 
mandated one Friday a month. Thus we bypass the problem of 
having to make ccanplex or expensive alternative plans to 
•cover* the classrooms of teachers pulled out for training," 
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Tests and Test Analysis 

The Iowa Test of Basic SUlls, the instrument of choice 
for the Cotopaxi project, vill be used to gather baseline data 
and to monitor the students throughout the project. It is ex- 
pected that the average grade equivalent score in reading and 
aath for all students will increase at least one grade- level 
for each year of the project. 

A secondary hypothesis relative to achievaaent is that 
the interventions will have an equal effect on students from 
different ability groups. To ascertain if this happens, stu- 
dents 1^11 be divided into abUity groups and their scores 
averaged and analyzed by group„ 

They also hypothesize that there will be a trend for stu- 
dents to move to higher ability groups throughout the project. 
To monitor this, evaluators will look for spikes or dips in the 
achievement ratings or jun^js for each group. The bi-weekly 
achievement ratings used to monitor the sti«lents will also pro- 
vide information as to the effects of the teacher workshops; 

Data on Engagement Rates 

Observations will be bi-weekly and two instruments will be 
used. One will chart engagement in academic activity, the 
other engagement in nonacademic activity. 

The observer who will be in class for a full day will 
record observations every five minutes for academic times and 
every ten minutes during activities related to other content 
areas. The school day will be divided into five periods and 
the average engagement rate will be calculated for all students 
in each of the five periods. 

To determine if there is an "interactive effect between 
student ability level and time of day," Cotopaxi will see if 
students of varying ability groups behave differently through- 
out the day. ^ 

Curriculum Analysis 

Since one of Cotopaxi 's c&jectlves is to improve curricu- 
lum, both the procedure used to analyze the existing curriculum 
and the process used to change it will be recorded. 
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Data on Teacher Behavior 

In order to detexxolne to what extent the workshops change 
teacher behavior in the areas of tine laaxiageinentf discipline, 
and classrodB structuring and organization: 

1. Teachers will be monitored bi-weekly^ 

2. They will receive reports and confer with their 
principals^ 

3. They will attend inscsrvice workshops, 

4. They will be monitored again. 



C. The Hapa CCalifornia) Project . 

Napa County's Professional Development Center devised a 
plan which draws on effective teacher and administrative prac- 
tices from a number of research projects. They hope, through 
their study, to identify the critical factors which are posi- 
tively relate to the improvement of instructional processes, 
academic learning time and student achievement. The project 
approach addresses three major areas: 

1. Staff Development 

2. Instructional Leadership 

3. Articulation TVround Instructional and Curriculum 
Issues 

According to Pam Robbins, Project Director, the plan calls 



• Teacher training which systematically roovec from the 
presentation of theory, to the demonstration of skills, 
to practice, to application of the practice in the 
classroom, to feedback about performance and in-class 
coaching, 

• "Directed* discussions between staff and local consult- 
ants Cspecial education teachers, reading specialists, 
county level curriculum consultants and Head Start per- 
sonnel) focused on the specialists' most effective 
classr( am management strategies to determine tasks ap- 
propriate to each child and encourage students to stay on 
task • 
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• Instxuctional Leadership Training which %fill, among 
other things, prepare principals to observe lessons, 
and analyze theiQ according to an established framework. 
Principals will be trained to look for such elements in 
t£» lesson as planning, motivation, teaching to objec- 
tSvj, task analysis, diagnosis and prescription, moni- 
toring student achievesmt and providing fe^ack to 
«e learners, as well as how the lessons comport with 
wliat we know about the affective and cognitive develop- 
ment of children. *^ 

The Hapa Plan is extremely ambitious, as evidenced by 
these sample questions that will be addressed by the evalua- 
tions 

input 

How is the program planned? By whom? What process? 
What goals? 

How is the program implentented? Training content, format, 
materials, personnel, coordination, activities? 

What changes w&ze made and why? 

What is the cost of the program? 

Site Characteristics 

What are the characteristics of the sites: i.e.. 

Community Cethnicity, rural/urban, etc.)? 

Teachers (experience, age, credentials, training)? 

Students (ability, LES, ethnicity, etc.)? 

Principal (experience, age, time at school, training)? 

How have the site characteristics, curricula, and sup- 
plemental programs changed over the four-ye^r period? 

Feelings of Participants 

What are the feelings of the participants toward the pro- 
gram? Developers, trainers, staff, principal, c-»mraunitv. 
student reaction/attitudes? 
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Priftcjfial Training Ef f ectlveaieas 

Hov effective is the principal dinJ^pal supervision training? 
Ability to take -anecdotal record of teacHing, analyze a teach-> 
ing episode, diagnose teaching and write objectives, select 
appropriate conference node, plan follow-up activities? 

Bovr effective is the principal team-building training? Plan- 
ning skills, decision making strategy selection, use of prob- 
lesax solving model, allocating appropriate resources? 

Staff Training Effectiveness 

Does articulation occur between parents and staff within and 
between schools and grade levels? 

Vlhat kinds of staff development have teachers had? 

Do train 3d staff members use the rcccamaended 8tr£«tegies for 
increasing ALT? 

Do trained staff members use the suggested classroom manage- 
ment strategies? 

Do trained staff members use the instructional processes that 
%^re taught in ths inservice workshops? 

Do teachers allocate more time to academic instruction? 

Pupil Outcomes 

Do teachers have greater overall xstep of pupil engagement and 
success rate than they did before the training began? 



Bas student attendance changed? 

Do students make greater yearly reading and math gains than they 
did before the training began? (In May of each year are 
achievement gains greater in the last two years?) 



Broad Effects 

What are the disadvantages and advantages of the program as seen 
by developers, implewenters , ttainets, staff and principals? 
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The total model is described by the following diagrfw whidi illustrates the reJaUcnship 

faefcMeen staff develc^raent, instructional leadership &nd articulation and the prime foous' 

engaged learning ti»e to Inprotre the achievanent students in the areas of reading wri'mathaaaUcs 



Adainistrator Training 



1. Sk^iervisicn 

2. Planning 

3. Decision Making 

4. Problon Solving 

5. Allocation of 
Be«3urces - 



Supervision 
and Tean 
BuiMing 



Staff Training 

1. Instructional 
Skills 

2. ClassrooB 
Msnageroent 
Skills 

3. Skills in 
Structuring 
Instructional 
time 



PlfiBBiiiiLj fcr 
^knstructioksal 
Pro(%sses 



Artiailaticn 
across grade levels 
and Between Schools 
(includes Head Start 



Classroom Instnictinna] 
Prcxijesses 

1. Diagnosis 
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aiKi Prevrious Perfonnance [ 



Student classroom learning 
(as shown by student 
behaviors — ALT 



Student I 
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The Napa approach, which ^phasizes staff development, 
instructional leadership and articulation, aims at pro- 
ducing the teacher behaviors identified by the Beginning 
Teacher Evaluation Study. The goals are to increase stu- 
dent learning tiaie and raise student achieveiaent scores. 
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The Instxu^nts 

Napa has found and developed numerous tools for collect- 
ing infarmatlon Including s 

• Extensive intervie%rs to ascertain teacher and adminis- 
trator feelings and to learn what they want out of the 
program; 

• The Comnanity Demographic Form and Student Demographic 
Form to provide data on the site characteristics and 
student characteristics; 

• Loc£lly«developed questionnaires to teachers to deter- 
mine ho» often they ar-i visited by their principals « 
and the mmber and nature of the folxow up conferences; 
to assess the organizational climate in the schools, 
and skills that staff have learned during training 
which they find most useful. 

In addition a number of observation instruments will be used 
such as The Time-on-Task Form, the Secondary Observation Instru- 
ment, Five-Minute Interaction, and Snapshot form developed by 
Jane S tailings. 

Napa will use the reading and math portions of the Stanford 
Achiev^nent Tests in one school and the California Achievaaent 
Test in the other. They are considering adding the Metropolitan 
Achievement Test but because one school has "matched" its cur- 
riculum to the MAT and the other school has not, the results may 
be inconclusive. 

Observation Schedule 

Each teacher will be observed with all the observation in- 
striunents in one^and two-hour segments during April and May each 
J year. In. all, each teacher and class will be observed for eight 

hours during the spring data collection period. 

Time-on-Task and implementation Will again be observed on 
two consecutive days in late October and late January - 
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D- The Oakland (California) Project ; 

•<We propose to improve student achievement through a 
staff developnent program focused on leadership , * said Marilyn 
Matsumoto Jones, principal investigatorl "Our goal is to 
strengthen those administrative and supervisory functions that 
cause schools to change." 

To accooiplish this, the leaders of each school (princi- 
pal, assistant principal, state and federal project director, 
reading ax^ math resource teachers, parent/community liaison, 
head teacher. School Advisory Council and other advisory com- 
mittee chairpersons and union representatives) will first be 
instructed in how increasing Academic Learning Time (ALT) can 
improve sti^dent learning. Other potent concepts will be drawn 
from the wlsdcmi that has emerged from studies of effective 
Inner-city schools and from studies of successful implementa- 
tion. 



"We want our school leaders to create new support systems 
for their teachers," said Ms. Jones. "Their teachers need 
oppcTtunities for collegial sharing, for supportive feedback 
and technical assistance. We don't mean criticism — we mean 
concrete suggestions that encourage adaptation of successful 
classroom practices. Teachers want to succeed — they want to 
feel that they are growing in competence and professionalism. 

"Over and above this work at the classroom level, we in- 
tend to take a 'whole school' approach to focusing on learning. 
Everyone involved will be .encouraged to discover %my8 in which 
the entire building can be managed better in order to get this 
focus „ For examples: what distractions do classroom teachers 
encounter in the cotirse of the day and how can they be avoided? 
Can the administrative routines within the building further 
strengthen learning? Can the schedule be more flexible? Is 
too much time being sp^nt on non- learning tasks? 

"We realize that there's no single answer for every teach- 
er. Some of theic will want to develop strategies to improve 
students' success rates, for instance, while others may prefer 
to focus on the use of new materials. But whatever strategies 
the individual teachers choose, they will all derive from the 
school-wide — and fervent commitment to improving learning . * 

In both of the participating schools the principals, re- 
source teachers and selected classroom teachers will plan staff 
developBtent workshops and observe K-4 teachers to help them 
perfect their classroom techniques. 
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"We see the assessment process for identifying the al- 
located tine, engaged time and success rates for reading and 
math now in place as a taking -of f point. With that infor- 
mation we can plan a staff development program that will 
be truly responsive to our local needs," said Ms. Jones. 

Many of Oakland's strategies such as documenting 
the progress of the program in both "naturalistic" and 
"scientific" nodes, involving adxcinistrative staff in 
workshops^ making it possible for teachers to monitor 
other teachers — parallel those already described for Detroit 

and Napa. But while Detroit is concentrating an teacher clev- 
elopinent directly, Oakland is banking on the effect a train- 
ing program for administrators will have on teachers. 

To this end, teachers and principals will work togeth- 
er to set goals, time-tables for tliose goais^ and determine 
what aspects of ALT they will tackle first and which next. 

Oakland is also still trying to work out a plan fr 
parent run howe-ectivity centers. In this plan, two 
parents from each of the schools would volunteer their homes and 
work with children after-school for one hour twice a week. 
The home-activity centers would accomodate approximately 
eight children for six week periods. Parents would te 
supervised by a Follow Through trained parent, and would 
teach the children in their charge games for reading and 
math skill development. 
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What Has Seen Learned To Date? 

The current Follow Through Pilot Project effort has just be- 
Thus far its impact on local staff has come frcsn the planning 
meetings rather than specific training in Academic Learning 
Time. But even the activities to date have stimulated staff. 
While none of the followina reports frc»n the field are astonish- 
ing, they are worth noting: 

• Each of the Technical Advisors (TAs) provided by the 
NIE received kudos from the districts. They lauded 
Gary Borich and Robert Floden for their extraordinary 
help in finding and designing evaluative 5 nstrimients, 
Yvonna Lincoln for being **not only brilliant and in- 
ventive in 'unobtrusive/naturalistic*- style information 
gathering, but also exceedingly knowledgeable about 
traditional evaluation and research, and willing to 
switch gears." They felt that Judith Little breathed 
life into the idea of collaborative effozt and David 
Berliner into the concept of ALT, Dennis Sparks "was 
the ideal catalyst." "His relationship to people was 
fantastic." "He really could bring the groups togeth- 
er ... he heard what others were saying and saw to it 
that others heard it too." 

None of the TAs \iBre considered superfluous. The only 
criticism came from one project director who felt that 
"their usefulness was limited by when they were available 
. , . districts would benefit from their involvement even 
after the project got officially started.** Several dis- 
tricts have arranged to engage some of the TAs on their 
own as needed. 

• Teachers in a number of districts have indicated that 
although they appreciate the support, which will make 
them better teachers, they feel it "will take a long 
time for the training to make up for the loss of a 
classroom aide." Displaced aides also were upset — until 
they were assured they had new placements. Parents 
regretted the loss of an aide in the classroom, until 
reassured. 

• Cotopaoci's teachers surprised their superintendent by 
demonstrating an unexpected enthusiasm, commitment and 
creativity in the planning process. Teachers who had not 
had the opportunity for workshops before were thrilled at 
the prospect of inservice training and the opportunity to 
observe other teachers, 

• Napa's introductory sessions were so effective in clari- 
fying cctations and the need for full commitment that 
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one principal bowed out of the program. His withdrawal 
testifies to the sensitivity of Napa's trainers who 
were able to elicit, early-on, the real reactions and 
doiibts of participants and conmninicate that. tixBy could 
be trusted not to be punitive if a participant was so 
uncomfortable with the program that he wished to 
leave. 

• As might be expected^ the teachers reactions to intro- 
ductory workshops and sample visits are idiosyncratic. 
Feedback in Cotopaxi rem the gamut from very positive 
reactions — e.g,, "i liked the tour of the classrooms 
and hearing other teachers* philosophies," and "Great 
relevancy to everyday nitty gritty in the classroom!" 
•Lots of new ideas to begin thinking," "Lots of insight 
to where I am and what I do in teaching — bringing 
concrete out of intuition," to "I feel the tours of 
classrocmis are too long, repetitive and very unprofit- 
able." By and large the responses were positive with 
frustration at having too little time to cover too much, 
or personal impatience — i.e., "I feel frustrated at 
wanting to be able to change and understand faster . " 

• "Most teachers are simply unaware of the minutes that 
escape from work. They have been accustcmied to think 
that a quiet room is a working room. They don't notice 
how they waste the children's learning time. "This proj- 
ect has made us all more sensitive to what a terrible 
waste it is to take seven minutes tr xite an assignment 
on the board while the children all »..iit 'quietly* and 
patiently," said Marilyn Jones from Oakland. 

« 

• While sorae districts intend to provide the trsiining 
and the support systems as a matter of course, Detroit 
has decided to provide clear targets and ask teachers 
to find the way to meet the expectations. "If teachers 
are spending over 5% of their time on management — 
it's too much. If less than 50% of their ime is on 
direct instruction — it's too little. And if students 
are engaged less than 80% of their academic time, reforms 
are needed," declared Sofer. "How teachers get the 
skills is their business as long as they keep improving 
until they meet the requirements." Naturally, Detroit 
has a prograra in place to help those who need it, but 

if a teacher is ali'aady meeting expectations 7 she could 
forgo the interventions. 



• Teachers in both Napa schools 
But after the "retreat," they 
with the goals of the pvoject 



were apprehensive at first, 
have become more familiar 
and, with great optimism. 
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have begun to make plans. Said one teacher, "We're ex- 
cited by the notion of being involved in a research ef- 
fort and doing everything we can to keep this alive. 
We knoif it*£s the planning year but we're anxious to get 
into it, " These sanje t -achers have set aside one 
schcx>l bulletin board tor posting reactions to reading, 
ideas about implementation and notes from their partner 
school regarding their progress and questions. 

"We're running into some difficulties with developing a 
sufficiently sensitive tool for observations, *• sadoi Oakland* 
Jones. "The problem we all have to face when we attempt 
to chart a teacher's reactions to children is that there 
are many non-verbal communications which just won't be 
picked up by an outside observer, but which the children 
understand full well. A n«i or a smile from one teacher 
means 'that's a great answer.* The words would be 
superfluous. And scsne teachers can glance over at a 
student in such a way as to demand attention — words 
are often unnecessary." 

Sheldon Sofer reports that in Detroit the spin-offs have 
been most heartening. A ninth grade reading program is 
interested in using their observational instruments to 
increase the awareness of teachers of their behavior, 
and to increase academic learning time. 

Several other Detroit elementary schools also have shown 
interest in replicating the mirrors and monitors proj- 
ect, although they will do so on their own, since they 
don't want to give up their classroom aides. 
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AFTERWORD 



Exectations can energize. Without them we might he 
content to drift along aimlessly as persons, and as a pro- 
fession. 

Teachers become powerful practitioners when emboldened 
by the expectations of their administrators . Children who, 
told repeatedly by their teachers that they can, and should — 
do! 

But presuppositions can stymie us — making progress less 
likely. In compensatory education one such coram cm presump- 
tion has been that quality education for disadvantaged young- 
sters has to have a high price tag. The Institute has challen- 
ged that belief by suggesting othe^ wise. This challenge nay 
be one of the most Important aspects of the Follow Through 
Pilot Projects as financial resources for the education of 
low^ income children gets scarcer. 

Other presuppositions that have been challenged by the 
work reported herein ares 

• There was the presumption on the part of some that the 
•feds' would "continue to develop impractical programs 
in»bouse and then seduce 'hungry' professionals to't»iy 
in. ' " But this time government people listened to 
practitioners and researchers in the field before develop- 
ing research guidelines. The result seems to be a 
pragmatic and affordable approach and a new expectation 
for collaborative goal setting. 
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• There was the presiamption that researchers and 
school people would continue to talk at or past each 
other. But Instead they talked with each other. 

And a new cucpectation emerged that a true conversation 
resulting in a workable amalgam of approaches can be the 
norm. 

• There was the presumption that most teachers would 
fight to keep their doors closed and their autonomy 
intact. But in fact, teachers who understand the 
benefits are opening their classrocan doors to sup- 
portive peers and principals. And if this new ex- 
pectation opens doors across the countr • and re- 
places isolation with wholesowa collabc ration, that 
will be another breakthrough. 

If the disadvantaged youngsters in these pilot schools 
becc^ne conpetent learners despite the cutback in funding , 
and the programs are sufficiently well-documented so they 
can be replicated by other communities, that will indeed be 
a most important milestone in the history of public educa- 
tion reform. 
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EXAMPLES 0? OBSERVATION FOR!!S USED BY PROJECTS 
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Student Off-Task SeatlOR Chart (co be collected every five minutes) 
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The purpose of this seating chart is to record each student who 
Is off -task at each five-olnute cl^ckpolnt. 

A seating chart Is coopleted every five-minutes for the length 
of the period. For exas^le, ten seating charts would be completed 
in a fifty minute period • At the top of each chart, the activity 
that is nccurrlng should be recorded • For example , "Teacher giving 
instruction from the chalkboard'% ""Students reading silently", or 
"Student reading aloud". It is possible to coioplete two time 
sequences on one form by using a black ink for the first recording 
and red ink for the second recording (sec figure 3). 

The observer will need to have a seating chart with the student's 
maxs recorded on it. ImtKdiately after the period starts » the 
observer scans the room going clockwise. Any student who is observed 
to be talking to another student will be marked with an "S", for 
Socializing, in the box under the sttident^s na^. For exaaq>le« Mary 
and Sue (in figure 3) were observed to be soclklizing. 

Any student who was observed to be uninvolved in^he teacher's 
expected acrivlty, foi: exa^le, staring out tUd 'window, out of their 
seatt or sharpening^ pencil, will be riarked wlt>tt for Uninvolved, 

under their names (se Harry and Sam in figure 3) . Uninvolved means the 
student is not speaking to another student but is clea*:ly not Involved 
with the lesson at hand. 

A student who is waiting for assistance will be coded with a "W", 
for Waiting, under their nai^ (see Frank in figure 3). 

A sus=&ary of the percent of students off -task can be found by 
using the following foramla: 

the sun of the number of students off -task for each observation 
the nucber of students x Uie. aumber of observations 

for example: 

In a classroom of thirty students, 10 observations were made. In 
the first observation, 2 students were observed to be off-task; in che 
second observation, 4 students were off -task; third « 3 students; 
fourth « 5; fifth * 3; sixth * 1; seventh « 2; eighth « 4; ninth « 7; 
and the tenth timet 6 students were off task. 

Using these figures, we obtain the following equation: 

2-HA-H3^"5-^3-Hl-h2'f4-f74>6 ^ 

30 X 10 300 ' 

Thus, we have found that 12.3% of the students were off-task during this 
period. 



NOTE: If the classroom seating takes a different form than the seating 
chart, for example, tables in a horseshoe formation instead of desks, 
then the seating chart should be corrected to couform to the actual 
classroom arrangement. The important thing is f; get each student's name 
in the right place on the seating chart. 
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ACTIVITY Uumiltu. UUJ^ 



ACTIVITY 2:-7Sk^ /m^^}^ 



STUDENTS OFF-TASK 



1 * ,Rav . 


* Row 


' Row 


* Row 


* Row 


* Row 


^ Row 


■ Row 


' Row 
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CODES 

S Socializing 

U « Uninvolved 

W Waiting for asiJ stance 

Complete a form each 5 !uiniit'-;S. 
Place students^ name in rows. 



For Exaiaple: 



John L 

IK- 
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